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Introduction 


In August 2010, the Smithsonian Science Education Center (SSEC), a division of the Smithsonian 
Institution formerly known as the National Science Resources Center, received a grant of more than $25 
million from the U.S. Department of Education’s Investing in Innovation (i3) program for a five-year 
study to validate its Leadership Assistance for Science Education Reform (LASER) model in three very 
diverse regions of the United States: rural North Carolina, northern New Mexico, and the Houston 
Independent School District (HISD). Matching funds to support the study in the amount of more than $5 
million were obtained from partners in the three regions as required by the Department of Education. 


The independent third-party research evaluation of this five-year validation study was conducted by the 
Center for Research in Educational Policy (CREP) with technical assistance from Westat and Abt 
Associates, who were provided to i3 grantees’ evaluation partners by the US Department of Education 
(USDOE). CREP, a Tennessee Center of Excellence, is a research and evaluation unit based at the College 
of Education, Health and Human Sciences at the University of Memphis. 


Interim evaluation results have been reported to SSEC by CREP in annual formal technical reports as well 
as more informally through presentations and written materials. This document presents overall 
findings, conclusions, and recommendations in summary form based on analysis of all years of available 
quantitative and qualitative data. Supporting materials comprising the complete final report include: 


e Anoverview of implementation findings related to the five pillars of the LASER model: 
O Research-based curriculum 
oO. Differentiated professional development 
O Administrative and community support 
O Materials support 
Oo Assessment 
e = Afinal technical report containing analyses of achievement data from all regions as well 
as the results of HLM analyses of data previously reported 
e Areport of findings from the case studies conducted in 20 of the participating schools 


Methodology 


The LASER i3 Validation study utilized a matched-pair, randomized controlled trial (RCT) complemented 
by multiple selected case studies. This design meets the What Works Clearinghouse criteria “without 
reservations,” which is the highest possible rating. 


Schools in the three regions with intact elementary (grades 3-5) and middle school (grades 6-8) cohorts 
were paired and randomly assigned to Phase 1 (immediate implementation) or Phase 2 (delayed 
implementation). The matched-pair design was utilized to ensure equivalency between groups. 
Implementation data were collected using surveys, structured observations, interviews, and focus 
groups. The evaluation team elected to use the WestEd developed Partnership for Standards-based 


Science Assessment (PASS) test as the primary measure of student learning. State specific achievement 
test scores were also analyzed. The study was conducted to answer four primary research questions: 


1. Dostudents in schools that receive the LASER model over three years (i.e., Phase 1 schools) 
attain higher levels of science achievement than students in schools that do not receive the 
LASER model (i.e., Phase 2 schools) as measured by: 

a. PASS-Basic (multiple choice questions only)? 
b. PASS-Extended (multiple choice questions plus open-ended and performance tasks)? 
c. State assessments? 

2. Dothese results vary by subgroup? 

3. What factors influence the implementation of the LASER model? 

4. What is the impact of participation in the LASER model on student attitudes toward science? 


Note: The data for the elementary and middle school samples reported in this manuscript should be 
treated as two separate studies for reporting and interpretation purposes--an elementary school study 
and a middle school study. Baseline equivalence was established for both samples. For the middle 
school study, please note that all sixth graders in the impact analyses were not in their schools at the 
time of random assignment of clusters, and therefore by definition, were considered "early" joiners 
(students not enrolled in the school at the time of random assignment). There were no joiners in the 
elementary study. Please also note that while the word "student" is used throughout this manuscript for 
ease in reporting, inferences for the middle school results should be at the "school" level. 


Population and Context 


The population from which the sample for this study was drawn encompasses three regions (Houston 
Independent School District, rural North Carolina, and northern New Mexico), and represents a total of 
16 school districts. This population of school districts includes more than 325,000 students and 20,000 
teachers, and over 150 district and building level instructional leaders, with more than half of students 
(56.2%) identified as “economically disadvantaged” by the National Center for Educational Statistics 
(NCES) based on free and reduced lunch status. From this total, schools were nominated for 
participation in the study, and from that nominated list the study sample was created. 


Although these regions, and schools within the regions, are very diverse, commonalities exist across the 
regions that stem from conversations, trends and initiatives taking place at the national level. These 
have had varying levels of impact on the teaching of science in elementary and middle schools and 
therefore on the implementation of the LASER model during the course of this study. Most notable are 
the national debate around Common Core State Standards and associated testing; the Next Generation 
Science Standards (final draft released in April 2013); new teacher evaluation models; and identification 
and implementation of programs for low performing schools. 


Within each region, unique conditions also existed during the LASER implementation window that had 
potential to impact implementation of the model. In North Carolina, a program called Read to Achieve 
was adopted in July 2012 via a state budget act and became effective during the 2013-2014 school year. 
With mandated summer reading camps and possible retention imposed on students who did not 
achieve acceptable reading levels by third grade, more instructional time was devoted to the teaching of 
reading in the lower grades in many NC schools. In Houston, eleven of the LASER i3 schools were also 
part of that district’s Apollo 20 initiative, with multiple strategies in place to close gaps in achievement 


and more time devoted to testing than was required in other schools. Implementation in Northern New 
Mexico was impacted by the geography of the region: remote mountain school locations affected 
teacher travel to professional development sessions; delivery of materials, particularly live specimens; 
and access to SSEC’s regional support system. Partnership with the Los Alamos National Laboratory 
(LANL) Foundation provided support for schools in that area. 


Baseline data collected from teachers prior to LASER i3 implementation in fall 2011 revealed that most 
students in all three regions received science instruction from their classroom teachers rather than from 
a science specialist (reported by more than 90% of teachers in all three regions) and that these teachers 
did not major in science or science education. (Approximately 10% of respondents from New Mexico 
and HISD and 8% from North Carolina reported holding majors in science.) Science laboratories were 
more prevalent in Houston schools than in the other regions, with nearly half (49.8%) of teachers 
reporting that their students went to labs to receive science instruction compared to 10.5% in North 
Carolina and 1.5% in New Mexico. Time devoted to the teaching of science was also greater in Houston, 
with teachers reporting an average of 3.3 hours per week of science instruction compared to slightly 
under 2 % hours in North Carolina and slightly over 2 hours in New Mexico. 


When asked at baseline about challenges associated with teaching science in elementary and middle 
schools, teachers’ responses were consistent across all three regions. The greatest challenge (reported 
as substantial or significant by 70% of NM teachers, 63% of NC teachers, and 59% of HISD teachers) was 
limited time for science instruction. Limited funds for purchasing equipment and supplies and more 
emphasis on English/language arts and mathematics than science instruction were also reported 
challenges in all regions. 


Sample 


The study sample originally included 139 schools across the three regions. These schools went through a 
nomination and qualification process, were matched based on several school-level demographic and 
achievement variables, and then randomly assigned to Phase 1 (immediate implementation) or Phase 2 
(delayed implementation). After random assignment, changes in school participation occurred within 
each region, and the final sample contained 125 study schools within the 16 districts and encompassed 
approximately 60,000 students, 1,900 teachers, and over 140 district administrators and principals. A 
subsample of 9,000 students in two cohorts (third and sixth graders in 2011-12) was followed 
longitudinally over the three years of the study, and a further subset of focal schools was randomly 
selected to participate in additional components of data collection. Additionally, a small number of 
schools was identified for participation in case study research based upon factors identified as unique or 
having potential for impacting program implementation and outcomes. A breakdown of the study 
schools by region, phase, and focal status as well as a detailed description of selection methodology has 
been provided in previous annual reports. 


HISD is the largest school district in the study. Participating schools generally serve Hispanic and African 
American populations (63.8% and 28.9%, respectively), with most students (88.1%) identified as eligible 
for free and reduced lunch. 


New Mexico LASER schools in the participating school districts range in size from 26 to 984 students. 
New Mexico districts serve mostly Hispanic, White, and American Indian/ Alaskan populations (48.0%, 


34.8%, and 11.7%, respectively), with over half of students (58.0%) qualifying for free and reduced lunch 
status. 


The sizes of LASER schools in the participating school districts in North Carolina (NC) range from 186 to 
930 students. Most of the districts serve primarily White and African American populations (60.7% and 
18.1%, respectively), with almost two-thirds of students (63.1%) identified as eligible for free or reduced 
lunch by the North Carolina Public Schools. 


Summary of Key Findings Related to 
Student Achievement 


Although student achievement gains as measured by traditional tests comprise only one component of a 
successful intervention, it is the single outcome of most interest to many constituencies. To obtain valid 
achievement outcome data, CREP researchers analyzed scores from only students in the elementary and 
middle school cohorts for whom PASS scores were available at both data points (baseline and spring 
2014). State achievement scores for this same group of students were also analyzed. It is important to 
note that all schools identified as Phase 1 are considered to be in the “treatment” group regardless of 
their level of implementation of the LASER model, and that fidelity of implementation varied widely 
across regions and across schools within regions. 


The following table of results reports favorable performance by the treatment group (Phase 1) 
compared to the delayed treatment/control group (Phase 2) on the three sections of the Partnership for 
Standards-based Science Assessment (PASS) and on a number of state assessments in reading, math, 
and/or science. Results are identified as being statistically significant (or unlikely to have occurred by 
chance) and/or as meeting the What Works Clearinghouse standard for being substantively important or 
educationally meaningful. Positive results on the PASS or on state tests were sometimes evidenced by 
overcoming a substantial deficit on the pretest; these results are also included in the table. 


PASS Structured Response (Multiple Choice) in Elementary Schools 


Substantively Phase 1 group overcame an 


Important/ educationally meaningful 
Statistically Educationally Phase 2 advantage on the 
Subgroup Region Significant Meaningful pretest 
English Language New V V 
Learners (ELL) Mexico 
New a 
Males Mexico 
Narth V 
Males Carolina 
Students identified as North V VY 
Special Education Carolina 


Substantively Phase 1 group overcame an 
Important/ educationally meaningful 
Statistically Educationally Phase 2 advantage on the 
Subgroup Region Significant Meaningful pretest 
All Regi 
All Students Combined aes v 
Combined 
All Students Combined pela v 
Carolina 
English Language All Regions Sf 
Learners (ELL) Combined 
English Language New V 
Learners (ELL) Mexico 
English Language North V 
Learners (ELL) Carolina 
Females yom v 
Carolina 
Students identified as ; 
. All Regions 
Economically amined v 
Disadvantaged 
Students identified as 
Economically HISD v 
Disadvantaged 
Students identified as New V 
Special Education Mexico 
Students identified as North Vv 
Special Education Carolina 


Substantively Phase 1 group overcame an 
Important/ educationally meaningful 
Statistically | Educationally Phase 2 advantage on the 
Subgroup Region Significant Meaningful pretest 
Students identified as North e 
Special Education Carolina 


PASS Performance Task in Elementary Schools 


Substantively Phase 1 group overcame an 
Important/ educationally meaningful 
Statistically Educationally Phase 2 advantage on the 
Subgroup Region Significant Meaningful pretest 
All Students Combined av REBIONS v 
Combined 
All Students Combined —_HISD v v 
: New 
All Students Combined ; v 
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English Language All Regions 7 2 
Learners (ELL) Combined 
English Language / 
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Special Education Combined 
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Special Education Mexico 
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Substantively 


Important/ 


Educationally 


Phase 1 group overcame an 
educationally meaningful 
Phase 2 advantage on the 


Subgroup Region Meaningful pretest 
All Students Combined “ neetons 
Combined 
English Language North V 
Learners (ELL) Carolina 
English Language All Regions V 
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All Regions < 
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Males Combined 
Students identified as : 
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Substantively Phase 1 group overcame an 


Important/ educationally meaningful 
Statistically Educationally Phase 2 advantage on the 
Subgroup Region Significant Meaningful pretest 
Females ene sank v 
Science 
Students 
HISD: STAAR 
identified as Sens v 
Special Education 
Students New Mexico: 
identified as Standards Based a 


Special Education Assessment 
Reading Test 


State Assessments: Fall 2011- Spring 2014: Middle Schools 


Substantively Phase 1 group overcame 
Important/ an educationally 
Statistically | Educationally meaningful Phase 2 

Subgroup Region Significant Meaningful advantage on the pretest 
Ai students HISD: Stanford Math v v 
Combined 
a Sees HISD: Stanford Reading v ov 
Combined 
All Students North Carolina: End-of- Pe 
Combined grade Math 
English New Mexico: Standards 
Language Based Assessment v 
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English 
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Learners (ELL) 
Females HISD: Stanford Math v v 
Females HISD: Stanford Reading v v 
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Males HISD: Stanford Math v v 
Students 
ecole: HISD: Stanford Math v v 
Economically 
Disadvantaged 
Students 
igendiiee a HISD: Stanford Reading v 
Economically 
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Students 
igeuuiled a2 HISD: Stanford Reading v 
Special 
Education 
Students 
jgentificgas HISD: STARR Math v 
Special 
Education 


Important and positive trends are evidenced in outcomes related to characteristics commonly agreed 
upon as most valued by employers. Both the open-ended and performance task sections of the PASS call 
upon students to communicate their knowledge in written form. They also engage students in activities 
associated with critical thinking and problem solving. These twenty-first century skills are associated 
with college and career readiness; it is therefore noteworthy that these are the areas of achievement in 
which Phase 1 students/schools excelled. It is also important to note that the underserved populations 
of economically disadvantaged and special needs students, as well as those for whom English is a second 
language seem to have benefited from their experiences with LASER, both as reflected in scores on the 
PASS and standardized tests of reading, mathematics, and science. 


Summary of Key Findings Related to 
Student Attitudes toward Science 


Qualitative data gathered through teacher focus groups, administrator interviews, and case studies 
indicate increased student engagement with and interest in science when kit-based inquiry strategies 
are used. Anecdotal data describe students looking forward to science activities, not wanting to be 
pulled out of science class for other services, and discussing science learnings with parents after school. 
Classroom observations support this finding, with consistently high student engagement levels recorded 
during times when the classroom components of the LASER model are being employed. 


Quantitative data regarding student attitudes toward science were gathered through a brief 
questionnaire administered at the same time as the PASS multiple choice test. The questionnaire 
contained demographic items and the following five questions related directly to student attitudes: 


| like science...(a lot, a little, or not at all) 

Do you think science will be useful to you when you are older? 

Would you like to be a scientist when you are older? 

How often to you talk to your family about what you do in science class? 
How often do you talk to your friends about what you do in science class? 


ae On 


Results showed few differences between students in Phase 1 and Phase 2 schools. It seems from the 
qualitative and anecdotal data that students were positively impacted by their experiences with kit- 
based inquiry science in ways that were not captured by the questionnaire. Perhaps inclusion of these 
questions with the PASS test was a factor. Additional research on this topic is needed to determine the 
actual level of impact LASER can have on student attitudes toward science. 


Summary of Key Findings Related to 
Implementation Factors 


Factors influencing implementation fidelity were documented primarily through qualitative and 
observational data. Administrator, teacher, and student attrition are factors that typically impact the 
sustainability of curricular changes in K-12 schools. In the case of LASER i3, many district and school 
leaders who committed to the work in 2010 were no longer in place by the end of the grant funded 
period. Ongoing support for all Phase 1 schools was provided by the Smithsonian Science Education 
Center for the duration of the grant; however, fidelity of implementation was directly impacted in some 
cases by changes in school and district leadership and shifts in priorities. Additionally, concerns about 
Common Core, changes in state tests, new teacher evaluations and Next Generation Science Standards 
have had variable levels of impact on the commitment of districts and schools to the teaching of science 
and on teachers’ perceptions of their ability to devote the time necessary to implement the LASER 
curriculum as designed. 


High quality professional development (PD) for science teachers is a central component of the LASER 
model. SSEC provided two levels of summer professional development for all science teachers in Phase 1 
schools: Level 1 sessions during the summer preceding the launch of each unit at each grade level to 
familiarize teachers with basic concepts and introduce the kits, and Level 2 sessions during the summer 
following the initial kit implementation to provide a “deeper dive” into the content associated with each 
unit. Condensed kit sessions were created for teachers unable to attend the summer sessions. Ideally, a 
teacher would be able to attend both Level 1 and Level 2 sessions for all three units provided for a single 
grade level, building both a strong science knowledge base and increased competence in inquiry 
learning strategies. 


Although devoting a week in the summer to PD was problematic for some, feedback on these sessions 
was universally positive. In fact, it is SSEC’s commitment to thoroughly preparing teachers to teach 
LASER’s inquiry-based units that reportedly sets this model apart from other kit-based science 
initiatives. Teachers who were able to attend all planned professional development sessions for a single 
grade level reported increased confidence, competence, and enthusiasm for teaching science using kits 
and inquiry strategies. However, due to attrition, only approximately 25% of teachers attending one 
session actually attended all three years. Additionally, one-fourth of that group changed grade levels 
during the implementation period. Teachers’ ability to take full advantage of the professional 
development opportunities offered had a direct impact on their capacity to implement the model in 
classrooms. 


LASER i3 was funded as a randomized control trial with standards set by the What Works Clearinghouse. 
Because of the need to maintain the sample size, level of implementation did not impact continued 
participation in the project and continued receipt of materials and other benefits. In addition, there was 
limited school- and regional-level ability to select units within grade level bands that were perceived to 
match differing standards and curriculum. Also, the study mandated a progression of unit 
implementation, with the expectation that three would be implemented at each grade level in the third 
year, leading to program fatigue in some schools and districts. 
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Conclusions 


Important and positive trends were found in achievement outcomes and in characteristics related to 
college and career readiness and commonly agreed upon as most valued by employers, as well as those 
directly related to science. These include written communication, critical thinking, and problem solving - 
skills that are essential to national competiveness going forward. This positive outcome should not be 
overlooked. It is also especially important to note that the underserved populations of economically 
disadvantaged and special needs students, as well as those for whom English is a second language, 
derived benefit from their experiences with LASER. 


Qualitative data gathered through teacher focus groups, administrator interviews, and case studies 
indicate increased student engagement with and interest in science when kit-based inquiry strategies 
are used. Observers saw much more experiential hands-on learning and cooperative/collaborative 
learning in classrooms that had access to the STC Units, and teachers in these classrooms reported 
greater preparedness to teach students to design and conduct experiments. 


In spite of current rhetoric regarding the need for increased emphasis in K-12 schools on all STEM 
(science, technology, engineering and mathematics) areas, the pressures associated with standardized 
tests has greatly decreased the time allocated to science teaching and learning in many of our nation’s 
K-12 public schools. Few schools participating in the current study allocated sufficient time at all grade 
levels to implement the science units as designed, and in some, science was taught only in the grade 
levels in which it was tested (usually 5° and 8" grades). Still, multiple positive outcomes were achieved, 
and much has been learned through this study regarding the potential of the LASER model, and 
particularly the use of inquiry-based strategies in elementary and middle school classrooms, to prepare 
students for success. 


Recommendations 


The Smithsonian Science Education Center was limited in its flexibility to alter the LASER project design 
during the course of this study due to the restrictions of the randomized control trial. The following 
recommendations for future implementations of LASER have been developed over time and have been 
discussed with SSEC: 


e Given the current educational climate and other priorities, it is difficult for most schools to 
implement three units with fidelity during school hours in a single academic year. Consider 
encouraging alternative scheduling, e.g. summer or after-school sessions to expose students to 
additional units. 

e Ensure initial and ongoing school-level commitment to LASER. District support is necessary but 
not sufficient for successful implementation. 

e Ensure that units align with Next Generation Science Standards and make this alignment explicit 
in marketing materials. 

e Consider creating a generic list of standards appropriate to each STC unit so that schools and 
districts can easily compare those with state standards driving the science curriculum. 
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Commit resources to creating Spanish editions of all materials to better meet the needs of ELL 
populations. 

Include formative assessments for incremental gauging of student understanding of science 
concepts central to each unit. 

Strengthen and make more obvious the links between science units and ELA/Math skills as well 
as “non-cognitive” skills such as perseverance, self-regulation, and cooperation. 

Increase the use of technology as a delivery system for teacher professional development as 
well as a support system for both teachers and students. Consider online or digital videos 
demonstrating appropriate ways to interact with each unit, possibly packaged as part of the 
teachers’ editions. 

Establish a certification process and database for teacher “experts” who can serve as regional 
resources. 

Focus resources on schools and districts with clear commitment to high-quality implementation 
so that outcomes can continue to be documented. 
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